Adaptive real-time anomaly detection with incremental clustering

Anomaly detection in information (IP) networks, detection of deviations from what is considered normal, is an important complement to misuse detection based on known attack descriptions. Performing anomaly detection in real-time places hard requirements on the algorithms used. First, to deal with the massive data volumes one needs to have efficient data structures and indexing mechanisms. Secondly, the dynamic nature of today’s information networks makes the characterisation of normal requests and services difficult. What is considered as normal during some time interval may be classified as abnormal in a new context, and vice versa. These factors make many proposed data mining techniques less suitable for real-time intrusion detection. In this paper we present ADWICE, Anomaly Detection With fast Incremental Clustering, and propose a new grid index that is shown to improve detection performance while preserving efficiency in search. Moreover, we propose two mechanisms for adaptive evolution of the normality model: incremental extension with new elements of normal behaviour, and a new feature that enables forgetting of outdated elements of normal behaviour. These address the needs of a dynamic network environment such as a telecom management network. We evaluate the technique for network-based intrusion detection, using the KDD data set as well as on data from a telecom IP test network. The experiments show good detection quality and act as proof of concept for adaptation of normality. a 2007 Elsevier Ltd. All rights reserved.